Awesome to see this PR coming in, Danny. I'm sure we'll be in touch as we review this code and as you continue to develop / finalize the code

Thanks for these contributions to CESM!

Thanks Will! Keep in touch :)

Hi @fvitt, @ekluzek and I would like to ask you a question:

• We noticed there are several soil erodibility files in the namelist_defaults_cam.xml for CAM4, CAM5, and CAM6. Should we move all of them to CTSM or should we just take the CAM6 one? Do you think people can actually use CAM4 but still have CTSM using the CAM6 erodibility file to calculate dust emissions?
• I am planning to copy all default values of the global tuning factor (dust_emis_fact) from namelist_defaults_cam.xml to namelist_defaults_ctsm.xml (maybe divided by 1.15). Does that sound good to you?

@dmleung I don't know which soil erod files should be moved to CTSM. @lkemmons and @tilmes may have an opinion on this.

I think, that there are people out there that still use different model configurations, like CAM4/WACCM4 (like Chuck for CARMA), and maybe CAM5 (I am not sure). I am not sure if people can run CAM4 using the CAM6 erodibility file, I don't know how those files differ from each other.

Okay, thank you @fvitt and @tilmes for the reply. I don't think the erod files used across different CAM versions are that different (the values differ slightly). I will look at the erod files again, think about it and discuss with people further.
@L3atm, please feel free to add any thoughts if helpful.

One thing to add here is the global dust tuning parameter to the CTSM namelist.

@dmleung Louisa @tilmes @wwieder @fvitt and I met to discuss this. We decided:

• The global dust tuning parameter dust_emis_fact will remain in CAM (although it's value will be expanded to take into account the dust emission method)
• We will make the CMEPS and then CTSM first, so this will all work together.

The issue about the dust emission method for CMEPS is now here:

ESCOMP/CMEPS#353

I've added a little design document about how to do this in CMEPS here:

https://docs.google.com/document/d/18nZ3LJF5W-YF9iBhqed6s_NWeKOvSSL2-k0Lye1nnLg

The design doc for the CMEPS shr_ part of this is in:

https://docs.google.com/document/d/18nZ3LJF5W-YF9iBhqed6s_NWeKOvSSL2-k0Lye1nnLg

I met with @dmleung and went over the current plans. I had proposed that CAM4, and CAM5 soil erodibility files be kept in CAM because of the effort needed to convert them to use in CTSM with streams. But, then he pointed out that he had already done this. So the files are already converted.

Here is the list of files in CAM right now...

<!-- soil erodibility factors -->
<soil_erod_file              >atm/cam/dst/dst_source2x2tunedcam6-2x2-04062017.nc </soil_erod_file>
<soil_erod_file phys="cam5"  >atm/cam/dst/dst_source2x2_cam5.4_c150327.nc</soil_erod_file>
<soil_erod_file phys="cam4"  >atm/cam/dst/dst_source2x2tuned-cam4-06132012.nc</soil_erod_file>
<soil_erod_file phys="cam4" hgrid="0.9x1.25">atm/cam/dst/dst_source1x1tuned-cam4-06202012.nc</soil_erod_file>

I can move these to CTSM, but need to use LND_TUNING_MODE to set whether CAM4, CAM5, or other version of CAM physics is being used. We currently don't have LND_TUNING_MODE settings for CAM4 or CAM5, but I can add them in. I will at the same time have to add those settings for the other fields that are tuned for CTSM, but will base them on the currently tuned values.

@ekluzek We can do either one of them, depending on which approach is easier for you and for the model. For record, the path to the converted soil erodibility files are in
/gpfs/fs1/work/dleung/Zender_dust_source

The stream file, src/cpl/share_esmf/ZenderSoilErodStreamType.F90, can process all soil erodibility files using the namelist variable stream_fldfilename_zendersoilerod.
I think we only need to add more entries in the bld/namelist_files/namelist_defaults_ctsm.xml.
Currently it contains the path to the CAM6 soil erodibility file

<stream_fldfilename_zendersoilerod
/gpfs/fs1/work/dleung/Zender_dust_source/cdf5_dst_source2x2tunedcam6-2x2-forCLM_c230311.nc</stream_fldfilename_zendersoilerod>

But we can add more entries for CTSM to use CAM5 and CAM4 soil erodibility files by detecting the CAM version (or physics) in an F case.

OK some strange things about testing with ne30 grids. If I run with too few processors it will fail with what I think are memory issues. With enough though it scales backwards so that more processors takes more overall run time, and 24 nodes causes it to hang. This is looking at total run time in the TestStatus file, so not accurate, but here it is...

SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:PASS SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=85
SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:PASS SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=105
SMS_D_P3072x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/TestStatus:FAIL SMS_D_P3072x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod RUN time=13275 (this hangs)

This is validated with the timing file, for model throughput. But, probably more testing needs to be done...

SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/timing/cesm_timing.SMS_D_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist.3607734.desched1.240228-152003: Model Throughput: 6.40 simulated_years/day
SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist/timing/cesm_timing.SMS_D_P1536x1_Ln9.ne30pg3_t232.I1850Clm51Sp.derecho_intel.clm-clm51cam6LndTuningModeZDustSoilErod.GC.ctsm51d169zenddelist.3684538.desched1.240301-155809: Model Throughput: 5.59 simulated_years/day

I thought maybe the problem with the ERI test was that the baselines had been messed up somehow. So I reran the baseline from ctsm5.1.dev169 and expected it to differ from the existing baselines. However, it compared exactly and showed no problems.

The difference in the test does seem like it's comparing different parts of the testing which would make it fail. But, I probably need to do more work to track down what it's doing.

Since, other tests are showing identical results. Including these ERI tests...

ERI_C2_Ld9.f10_f10_mg37.I2000Clm51BgcCrop.derecho_gnu.clm-default
ERI_D.ne30pg3_t232.I1850Clm51BgcCrop.derecho_intel.clm-clm51cam6LndTuningMode
ERI_D_Ld9.f10_f10_mg37.I1850Clm45Bgc.derecho_gnu.clm-default
ERI_D_Ld9.f10_f10_mg37.I1850Clm51Bgc.derecho_gnu.clm-default
ERI_D_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-default
ERI_D_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-default
ERI_D_Ld9.ne30_g17.I2000Clm50BgcCru.derecho_intel.clm-vrtlay
ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-default
ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-drydepnomegan
ERI_Ld9.f10_f10_mg37.I2000Clm50BgcCru.derecho_intel.clm-default
ERI_Ld9.f45_g37.I2000Clm50BgcCru.derecho_intel.clm-nofire

I'm also wondering if this is something that we can ignore as an odd glitch...

I was having trouble with the following test on izumi:

SMS_Ln9.f10_f10_mg37.I2000Clm45Sp.izumi_gnu.clm-clm45cam4LndTuningModeZDustSoilErod

It turned out that running with NAG found a legit problem!

diff --git a/src/cpl/share_esmf/ZenderSoilErodStreamType.F90 b/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
index 1755a32f7..0d5a7653d 100644
--- a/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
+++ b/src/cpl/share_esmf/ZenderSoilErodStreamType.F90
@@ -90,8 +90,8 @@ subroutine Init(this, bounds, NLFilename)
    character(len=*), parameter    :: stream_name = 'zendersoilerod'
    !-----------------------------------------------------------------------
 
-      call this%InitAllocate( bounds )
       call control%ReadNML( bounds, NLFileName )
+      call this%InitAllocate( bounds )
 
       if ( this%useStreams() )then     ! is this a namelist input and is it set in namelist default

Without this change an array doesn't get properly allocated! So this was a critical fix and it only showed up with the Nag compiler on izumi!

The cesm.log file showed the following:

Runtime Error: /fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/share_esmf/ZenderSoilErodStreamType.F90, line 160: Subscript 1 of THIS%SOIL_ERODIBILITY (value 1) is out of range (1:0)
Program terminated by fatal error
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/share_esmf/ZenderSoilErodStreamType.F90, line 160: Error occurred in ZENDERSOILERODSTREAMTYPE:INIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/biogeochem/DUSTMod.F90, line 90: Called by DUSTMOD:INIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/main/clm_instMod.F90, line 353: Called by CLM_INSTMOD:CLM_INSTINIT
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/main/clm_initializeMod.F90, line 406: Called by CLM_INITIALIZEMOD:INITIALIZE2
/fs/cgd/data0/erik/ctsm_worktree/zendersource/src/cpl/nuopc/lnd_comp_nuopc.F90, line 659: Called by LND_COMP_NUOPC:INITIALIZEREALIZE
/fs/cgd/data0/erik/ctsm_worktree/zendersource/components/cmeps/cime_config/../cesm/driver/esmApp.F90, line 128: Called by ESMAPP

@briandobbins this is an example of how using the Nag compiler can help us with our code.

I realized the continuing to fail ERI test is because of the finidat/use_init_interp settings that apparently was only showing up for this one test. Some changes in namelist_defaults_ctsm.xml should allow it to work.